{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 🪙 American Express - Default Prediction Competition Original Notebook\n",
    "![](./images/background.jpg)\n",
    "\n",
    "---\n",
    "\n",
    "In this [Kaggle competition](https://www.kaggle.com/competitions/g-research-crypto-forecasting/overview), you'll use your machine learning expertise to predict credit default. This competition is hosted by American Express. \n",
    "\n",
    "> American Express is a globally integrated payments company. The largest payment card issuer in the world, they provide customers with access to products, insights, and experiences that enrich lives and build business success.\n",
    "\n",
    "The dataset provided is an industrial scale data set of about 5.5 million rows. It has been pre-processed and converted to a lightweight version by raddar for ease of training and better result. This dataset is available in a [parquet format][1].\n",
    "\n",
    "[1]: https://www.kaggle.com/datasets/raddar/amex-data-integer-dtypes-parquet-format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Install necessary packages\n",
    "\n",
    "We can install the necessary package by either running pip install --user <package_name> or include everything in a requirements.txt file and run pip install --user -r requirements.txt. We have put the dependencies in a requirements.txt file so we will use the former method.\n",
    "\n",
    "NOTE: After installing python packages, restart notebook kernel before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install -r requirements.txt --user --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Imports\n",
    "\n",
    "In this section we import the packages we need for this example. Make it a habit to gather your imports in a single place. It will make your life easier if you are going to transform this notebook into a Kubeflow pipeline using Kale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "imports"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os, subprocess\n",
    "import random, zipfile, joblib\n",
    "import scipy.stats\n",
    "import warnings\n",
    "import gc, wget\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from lightgbm import LGBMClassifier, log_evaluation\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Project hyper-parameters\n",
    "\n",
    "In this cell, we define the different hyper-parameters. Defining them in one place makes it easier to experiment with their values and also facilitates the execution of HP Tuning experiments using Kale and Katib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "pipeline-parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Hyper-parameters\n",
    "N_EST = 30\n",
    "LR = 0.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Set random seed for reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "def fix_all_seeds(seed):\n",
    "    np.random.seed(seed)\n",
    "    random.seed(seed)\n",
    "    os.environ['PYTHONHASHSEED'] = str(seed)\n",
    "\n",
    "fix_all_seeds(2022)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Download data\n",
    "\n",
    "In this section, we download the data from kaggle using the Kaggle API credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": [
     "block:download_data"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['kaggle', 'datasets', 'download', '-d', 'raddar/amex-data-integer-dtypes-parquet-format'], returncode=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# setup kaggle environment for data download\n",
    "dataset = \"amex-data-integer-dtypes-parquet-format\"\n",
    "\n",
    "# setup kaggle environment for data download\n",
    "with open('/secret/kaggle-secret/password', 'r') as file:\n",
    "    kaggle_key = file.read().rstrip()\n",
    "with open('/secret/kaggle-secret/username', 'r') as file:\n",
    "    kaggle_user = file.read().rstrip()\n",
    "\n",
    "os.environ['KAGGLE_USERNAME'], os.environ['KAGGLE_KEY'] = kaggle_user, kaggle_key\n",
    "\n",
    "# download kaggle's Amex-credit-prediction data\n",
    "subprocess.run([\"kaggle\",\"datasets\", \"download\", \"-d\", f'raddar/{dataset}'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "block:"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CompletedProcess(args=['rm', 'data/train_labels.zip'], returncode=0)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to download to\n",
    "data_path = 'data'\n",
    "\n",
    "# extract Amex-credit-prediction.zip to data_path\n",
    "with zipfile.ZipFile(f\"{dataset}.zip\",\"r\") as zip_ref:\n",
    "    zip_ref.extractall(data_path)\n",
    "    \n",
    "# download kaggle's Amex-credit-prediction train_labels.zip\n",
    "download_link = \"https://github.com/kubeflow/examples/blob/master/american-express-default-kaggle-competition/data/train_labels.zip?raw=true\"\n",
    "wget.download(download_link, f'{data_path}/train_labels.zip')\n",
    "\n",
    "# extract Amex-credit-prediction.zip to data_path\n",
    "with zipfile.ZipFile(f'{data_path}/train_labels.zip','r') as zip_ref:\n",
    "    zip_ref.extractall(data_path)\n",
    "    \n",
    "# delete zipfiles\n",
    "subprocess.run(['rm', f'{dataset}.zip'])\n",
    "subprocess.run(['rm', f'{data_path}/train_labels.zip'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Load the dataset\n",
    "\n",
    "First, let us load and analyze the data.\n",
    "\n",
    "The data is in csv format, thus, we use the handy read_csv pandas method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": [
     "block:load_data",
     "prev:download_data"
    ]
   },
   "outputs": [],
   "source": [
    "TRAIN_CSV = (f'{data_path}/train.parquet')\n",
    "TEST_CSV = f'{data_path}/test.parquet'\n",
    "TARGET_CSV = f'{data_path}/train_labels.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target shape: (458913,)\n"
     ]
    }
   ],
   "source": [
    "df_train = pd.read_parquet(TRAIN_CSV)\n",
    "df_test = pd.read_parquet(TEST_CSV)\n",
    "target = pd.read_csv(TARGET_CSV).target.values\n",
    "print(f\"target shape: {target.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5531451, 190)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "customer_ID          0\n",
       "S_2                  0\n",
       "P_2              45985\n",
       "D_39                 0\n",
       "B_1                  0\n",
       "                ...   \n",
       "D_141           101548\n",
       "D_142          4587043\n",
       "D_143                0\n",
       "D_144            40727\n",
       "D_145                0\n",
       "Length: 190, dtype: int64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Define Helper Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": [
     "functions"
    ]
   },
   "outputs": [],
   "source": [
    "# @yunchonggan's fast metric implementation\n",
    "# From https://www.kaggle.com/competitions/amex-default-prediction/discussion/328020\n",
    "def amex_metric(y_true: np.array, y_pred: np.array) -> float:\n",
    "\n",
    "    # count of positives and negatives\n",
    "    n_pos = y_true.sum()\n",
    "    n_neg = y_true.shape[0] - n_pos\n",
    "\n",
    "    # sorting by descring prediction values\n",
    "    indices = np.argsort(y_pred)[::-1]\n",
    "    preds, target = y_pred[indices], y_true[indices]\n",
    "\n",
    "    # filter the top 4% by cumulative row weights\n",
    "    weight = 20.0 - target * 19.0\n",
    "    cum_norm_weight = (weight / weight.sum()).cumsum()\n",
    "    four_pct_filter = cum_norm_weight <= 0.04\n",
    "\n",
    "    # default rate captured at 4%\n",
    "    d = target[four_pct_filter].sum() / n_pos\n",
    "\n",
    "    # weighted gini coefficient\n",
    "    lorentz = (target / n_pos).cumsum()\n",
    "    gini = ((lorentz - cum_norm_weight) * weight).sum()\n",
    "\n",
    "    # max weighted gini coefficient\n",
    "    gini_max = 10 * n_neg * (1 - 19 / (n_pos + 20 * n_neg))\n",
    "\n",
    "    # normalized weighted gini coefficient\n",
    "    g = gini / gini_max\n",
    "\n",
    "    return 0.5 * (g + d)\n",
    "\n",
    "def lgb_amex_metric(y_true, y_pred):\n",
    "    \"\"\"The competition metric with lightgbm's calling convention\"\"\"\n",
    "    return ('amex_metric_score',\n",
    "            amex_metric(y_true, y_pred),\n",
    "            True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "features_avg = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_8', 'B_9', 'B_10', 'B_11', 'B_12', 'B_13', 'B_14', 'B_15', \n",
    "                'B_16', 'B_17', 'B_18', 'B_19', 'B_20', 'B_21', 'B_22', 'B_23', 'B_24', 'B_25', 'B_28', 'B_29', 'B_30', \n",
    "                'B_32', 'B_33', 'B_37', 'B_38', 'B_39', 'B_40', 'B_41', 'B_42', 'D_39', 'D_41', 'D_42', 'D_43', 'D_44', \n",
    "                'D_45', 'D_46', 'D_47', 'D_48', 'D_50', 'D_51', 'D_53', 'D_54', 'D_55', 'D_58', 'D_59', 'D_60', 'D_61', \n",
    "                'D_62', 'D_65', 'D_66', 'D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_75', 'D_76', 'D_77', 'D_78', \n",
    "                'D_80', 'D_82', 'D_84', 'D_86', 'D_91', 'D_92', 'D_94', 'D_96', 'D_103', 'D_104', 'D_108', 'D_112', 'D_113', \n",
    "                'D_114', 'D_115', 'D_117', 'D_118', 'D_119', 'D_120', 'D_121', 'D_122', 'D_123', 'D_124', 'D_125', 'D_126', \n",
    "                'D_128', 'D_129', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', 'D_140', 'D_141', 'D_142', 'D_144', \n",
    "                'D_145', 'P_2', 'P_3', 'P_4', 'R_1', 'R_2', 'R_3', 'R_7', 'R_8', 'R_9', 'R_10', 'R_11', 'R_14', 'R_15', 'R_16', \n",
    "                'R_17', 'R_20', 'R_21', 'R_22', 'R_24', 'R_26', 'R_27', 'S_3', 'S_5', 'S_6', 'S_7', 'S_9', 'S_11', 'S_12', 'S_13', \n",
    "                'S_15', 'S_16', 'S_18', 'S_22', 'S_23', 'S_25', 'S_26']\n",
    "features_min = ['B_2', 'B_4', 'B_5', 'B_9', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', 'B_19', 'B_20', 'B_28', 'B_29', 'B_33', 'B_36', \n",
    "                'B_42', 'D_39', 'D_41', 'D_42', 'D_45', 'D_46', 'D_48', 'D_50', 'D_51', 'D_53', 'D_55', 'D_56', 'D_58', 'D_59', \n",
    "                'D_60', 'D_62', 'D_70', 'D_71', 'D_74', 'D_75', 'D_78', 'D_83', 'D_102', 'D_112', 'D_113', 'D_115', 'D_118', 'D_119', \n",
    "                'D_121', 'D_122', 'D_128', 'D_132', 'D_140', 'D_141', 'D_144', 'D_145', 'P_2', 'P_3', 'R_1', 'R_27', 'S_3', 'S_5', \n",
    "                'S_7', 'S_9', 'S_11', 'S_12', 'S_23', 'S_25']\n",
    "features_max = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_7', 'B_8', 'B_9', 'B_10', 'B_12', 'B_13', 'B_14', 'B_15', 'B_16', 'B_17', \n",
    "                'B_18', 'B_19', 'B_21', 'B_23', 'B_24', 'B_25', 'B_29', 'B_30', 'B_33', 'B_37', 'B_38', 'B_39', 'B_40', 'B_42', 'D_39', \n",
    "                'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', 'D_48', 'D_49', 'D_50', 'D_52', 'D_55', 'D_56', 'D_58', 'D_59', \n",
    "                'D_60', 'D_61', 'D_63', 'D_64', 'D_65', 'D_70', 'D_71', 'D_72', 'D_73', 'D_74', 'D_76', 'D_77', 'D_78', 'D_80', 'D_82', \n",
    "                'D_84', 'D_91', 'D_102', 'D_105', 'D_107', 'D_110', 'D_111', 'D_112', 'D_115', 'D_116', 'D_117', 'D_118', 'D_119', \n",
    "                'D_121', 'D_122', 'D_123', 'D_124', 'D_125', 'D_126', 'D_128', 'D_131', 'D_132', 'D_133', 'D_134', 'D_135', 'D_136', \n",
    "                'D_138', 'D_140', 'D_141', 'D_142', 'D_144', 'D_145', 'P_2', 'P_3', 'P_4', 'R_1', 'R_3', 'R_5', 'R_6', 'R_7', 'R_8', \n",
    "                'R_10', 'R_11', 'R_14', 'R_17', 'R_20', 'R_26', 'R_27', 'S_3', 'S_5', 'S_7', 'S_8', 'S_11', 'S_12', 'S_13', 'S_15', 'S_16', \n",
    "                'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']\n",
    "features_last = ['B_1', 'B_2', 'B_3', 'B_4', 'B_5', 'B_6', 'B_7', 'B_8', 'B_9', 'B_10', 'B_11', 'B_12', 'B_13', 'B_14', 'B_15', 'B_16', \n",
    "                 'B_17', 'B_18', 'B_19', 'B_20', 'B_21', 'B_22', 'B_23', 'B_24', 'B_25', 'B_26', 'B_28', 'B_29', 'B_30', 'B_32', 'B_33', \n",
    "                 'B_36', 'B_37', 'B_38', 'B_39', 'B_40', 'B_41', 'B_42', 'D_39', 'D_41', 'D_42', 'D_43', 'D_44', 'D_45', 'D_46', 'D_47', \n",
    "                 'D_48', 'D_49', 'D_50', 'D_51', 'D_52', 'D_53', 'D_54', 'D_55', 'D_56', 'D_58', 'D_59', 'D_60', 'D_61', 'D_62', 'D_63', \n",
    "                 'D_64', 'D_65', 'D_69', 'D_70', 'D_71', 'D_72', 'D_73', 'D_75', 'D_76', 'D_77', 'D_78', 'D_79', 'D_80', 'D_81', 'D_82', \n",
    "                 'D_83', 'D_86', 'D_91', 'D_96', 'D_105', 'D_106', 'D_112', 'D_114', 'D_119', 'D_120', 'D_121', 'D_122', 'D_124', 'D_125', \n",
    "                 'D_126', 'D_127', 'D_130', 'D_131', 'D_132', 'D_133', 'D_134', 'D_138', 'D_140', 'D_141', 'D_142', 'D_145', 'P_2', 'P_3', \n",
    "                 'P_4', 'R_1', 'R_2', 'R_3', 'R_4', 'R_5', 'R_6', 'R_7', 'R_8', 'R_9', 'R_10', 'R_11', 'R_12', 'R_13', 'R_14', 'R_15', \n",
    "                 'R_19', 'R_20', 'R_26', 'R_27', 'S_3', 'S_5', 'S_6', 'S_7', 'S_8', 'S_9', 'S_11', 'S_12', 'S_13', 'S_16', 'S_19', 'S_20', \n",
    "                 'S_22', 'S_23', 'S_24', 'S_25', 'S_26', 'S_27']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "block:feature_engineering",
     "prev:load_data"
    ]
   },
   "outputs": [],
   "source": [
    "# feature engineering gotten from https://www.kaggle.com/code/ambrosm/amex-lightgbm-quickstart\n",
    "def get_features(df, \n",
    "                 features_avg, \n",
    "                 features_min, \n",
    "                 features_max, \n",
    "                 features_last\n",
    "                ):\n",
    "    '''\n",
    "    This function takes a dataframe with all features and returns the aggregated feature grouped by the customer id.\n",
    "    \n",
    "    df - dataframe\n",
    "    '''\n",
    "    cid = pd.Categorical(df.pop('customer_ID'), ordered=True) # get customer id\n",
    "    last = (cid != np.roll(cid, -1)) # mask for last statement of every customer\n",
    "   \n",
    "    df_avg = (df\n",
    "              .groupby(cid)\n",
    "              .mean()[features_avg]\n",
    "              .rename(columns={f: f\"{f}_avg\" for f in features_avg})\n",
    "             ) \n",
    "    \n",
    "    df_min = (df\n",
    "              .groupby(cid)\n",
    "              .min()[features_min]\n",
    "              .rename(columns={f: f\"{f}_min\" for f in features_min})\n",
    "             )\n",
    "    gc.collect()\n",
    "    print('Computed min')\n",
    "    \n",
    "    df_max = (df\n",
    "              .groupby(cid)\n",
    "              .max()[features_max]\n",
    "              .rename(columns={f: f\"{f}_max\" for f in features_max})\n",
    "             )\n",
    "    gc.collect()\n",
    "    print('Computed max')\n",
    "    \n",
    "    df = (df.loc[last, features_last]\n",
    "          .rename(columns={f: f\"{f}_last\" for f in features_last})\n",
    "          .set_index(np.asarray(cid[last]))\n",
    "         )\n",
    "    gc.collect()\n",
    "    print('Computed last')\n",
    "    \n",
    "    df_ = pd.concat([df, df_min, df_max, df_avg], axis=1, )\n",
    "    \n",
    "    del df, df_avg, df_min, df_max, cid, last\n",
    "    \n",
    "    return df_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computed min\n",
      "Computed max\n",
      "Computed last\n",
      "Computed min\n",
      "Computed max\n",
      "Computed last\n"
     ]
    }
   ],
   "source": [
    "# apply feature engineering function\n",
    "train = get_features(df_train, features_avg, features_min, features_max, features_last)\n",
    "test = get_features(df_test, features_avg, features_min, features_max, features_last)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "B_1_last    False\n",
       "B_2_last     True\n",
       "B_3_last     True\n",
       "B_4_last    False\n",
       "B_5_last    False\n",
       "            ...  \n",
       "S_18_avg    False\n",
       "S_22_avg     True\n",
       "S_23_avg     True\n",
       "S_25_avg     True\n",
       "S_26_avg    False\n",
       "Length: 469, dtype: bool"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check null values\n",
    "train.isna().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Modelling: StratifiedKFold\n",
    "\n",
    "We cross-validate with a six-fold StratifiedKFold to handle the imbalanced nature of the target.\n",
    "\n",
    "Lightgbm handles null values efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "block:modelling",
     "prev:feature_engineering"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "469 features\n",
      "[20]\tvalid_0's binary_logloss: 0.267976\tvalid_0's amex_metric_score: 0.750976\n",
      "Score = 0.7604229987279087\n",
      "Fold 0\n",
      "[20]\tvalid_0's binary_logloss: 0.267339\tvalid_0's amex_metric_score: 0.753257\n",
      "Score = 0.7624180573803372\n",
      "Fold 1\n",
      "OOF Score:                       0.76142\n"
     ]
    }
   ],
   "source": [
    "# Cross-validation\n",
    "\n",
    "features = [f for f in train.columns if f != 'customer_ID' and f != 'target']\n",
    "\n",
    "print(f\"{len(features)} features\")\n",
    "\n",
    "score_list = [] # lgbm score per fold\n",
    "y_pred_list = [] # fold predictions list\n",
    "\n",
    "# init StratifiedKFold\n",
    "kf = StratifiedKFold(n_splits=4)\n",
    "\n",
    "for fold, (idx_tr, idx_va) in enumerate(kf.split(train, target)):\n",
    "    \n",
    "    X_tr, X_va, y_tr, y_va, model = None, None, None, None, None\n",
    "\n",
    "    X_tr = train.iloc[idx_tr][features]\n",
    "    X_va = train.iloc[idx_va][features]\n",
    "    y_tr = target[idx_tr]\n",
    "    y_va = target[idx_va]\n",
    "    \n",
    "    # init model\n",
    "    model = LGBMClassifier(n_estimators=N_EST,\n",
    "                          learning_rate=LR, \n",
    "                          random_state=2022)\n",
    "    # fit model\n",
    "    model.fit(X_tr, y_tr,\n",
    "              eval_set = [(X_va, y_va)], \n",
    "              eval_metric=[lgb_amex_metric],\n",
    "              early_stopping_rounds=30,\n",
    "              callbacks=[log_evaluation(20)])\n",
    "    \n",
    "    X_tr, y_tr = None, None\n",
    "    \n",
    "    # fold validation set predictions\n",
    "    y_va_pred = model.predict_proba(X_va, raw_score=True)\n",
    "    \n",
    "    # model score\n",
    "    score = amex_metric(y_va, y_va_pred)\n",
    "\n",
    "    print(f\"Score = {score}\")\n",
    "    score_list.append(score)\n",
    "    \n",
    "    # test set predictions\n",
    "    y_pred_list.append(model.predict_proba(test[features], raw_score=True))\n",
    "    \n",
    "    print(f\"Fold {fold}\") \n",
    "\n",
    "# save model\n",
    "joblib.dump(model, 'lgb.jl')\n",
    "print(f\"OOF Score:                       {np.mean(score_list):.5f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABLMAAAI/CAYAAACMIJv7AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABda0lEQVR4nO3dfbTdZXnn//eHI5LQkNBK8EcBDShjKk9RN6BQFFELNtSCpWqnWrEPGaZQfIAOsc5U7LTToA6UisoEKVQKlYqiCAp2HGkpVeQEEkIAq0AspCqVh0gkoITr98f+hh4P52Gfx713eL/Wysre9/f63vf1Ze1luq5e9/1NVSFJkiRJkiT1g+26nYAkSZIkSZLUKYtZkiRJkiRJ6hsWsyRJkiRJktQ3LGZJkiRJkiSpb1jMkiRJkiRJUt+wmCVJkiRJkqS+8axuJ9Dvdtlll1q0aFG305AkSZIkSdpmrFq16gdVtXCkaxazpmjRokUMDg52Ow1JkiRJkqRtRpLvjHbNbYaSJEmSJEnqGxazJEmSJEmS1DcsZkmSJEmSJKlveGbWFK3dsJFFy6/udhqSJEmSJOkZav2Kpd1OYVb1TGdWki1JVie5Lcmnk+w4StyeSb6a5PYk65K8c5x5L0py/CTyOSLJoRO9T5IkSZIkSTOnZ4pZwOaqWlJV+wE/Bk4cJe4J4NSqejHwcuCkJC+egXyOACxmSZIkSZIk9ZBeKmYNdT3wwpEuVNV3q+rm5vMjwB3A7p1MmuSPk9zUdH+tTJJm/JSm0+vWJJ9Ksoh2Me3dTbfY4dPxUJIkSZIkSZqanjszK8mzgNcD13QQuwh4CXBjh9OfW1V/0tx7MXAM8AVgObBXVT2eZOeqejjJecCmqvrwCOsuA5YBDMxf2OHSkiRJkiRJmqpe6syam2Q1MAj8K3DBWMFJ5gGfAd5VVT/scI1XJ7kxyVrgSGDfZvxW4JIkb6W9jXFMVbWyqlpV1RrYcUGHS0uSJEmSJGmqeqkza3NVLekkMMn2tAtZl1TVZzu8Zw7wMaBVVfcmOQOY01xeCrwS+BXgfUn2n2DukiRJkiRJmgW91JnVkeacqwuAO6rqrAncurVw9YOmq+v4Zr7tgD2r6qvA6cACYB7wCLDTtCUuSZIkSZKkKeu7YhZwGPA24MjmcPbVSX55vJuq6mHgfOA24FrgpubSAPA3zdbDW4C/bGK/ABznAfCSJEmSJEm9I1XV7Rz6WqvVqsHBwW6nIUmSJEmStM1IsqqqWiNd68fOLEmSJEmSJD1D9dIB8D8lyXOAr4xw6TVV9cAI8R+lvQVxqHOq6sKZyE+SJEmSJEmzr2eLWU3BaskE4k+auWwkSZIkSZLUC9xmKEmSJEmSpL5hMUuSJEmSJEl9w2KWJEmSJEmS+obFLEmSJEmSJPUNi1mSJEmSJEnqGz37NsN+sXbDRhYtv7rbaUiSJEnST1m/Ymm3U5CkGTFrnVlJtiRZnWRdkjVJTk0y6vpJDm7iVzfxxw259s4ktzVzvWucdS9Kcvwk8j0iyaETvU+SJEmSJEkzZzY7szZX1RKAJLsClwLzgfePEn8b0KqqJ5LsBqxJ8gVgMfB7wMHAj4FrklxVVd+e5nyPADYB/zzN80qSJEmSJGmSunJmVlXdDywDTk6SUWIeraonmq9zgGo+/wJw45Dr/wC8sZN1k/xxkpuarq6VW9dOckqS25PcmuRTSRYBJwLvbjrDDp/0w0qSJEmSJGnadO0A+Kq6GxgAdh0tJskhSdYBa4ETm+LVbcDhSZ6TZEfgl4E9O1z23Ko6qKr2A+YCxzTjy4GXVNUBzTrrgfOAs6tqSVVdPyyvZUkGkwxueXRjx88sSZIkSZKkqenptxlW1Y1VtS9wEPDeJHOq6g7gTODLwDXAamBLh1O+OsmNSdYCRwL7NuO3ApckeSvwxKh3/0deK6uqVVWtgR0XTOyhJEmSJEmSNGldK2Yl2Zt2Eer+8WKbAtYmYL/m+wVV9bKqeiXwEPAvHaw3B/gYcHxV7Q+cT3v7IsBS4KPAS4GbkviWR0mSJEmSpB7UlWJWkoW0t/GdW1U1SsxeW4tKSZ5P++D39c33XZu/n0f7vKxLO1h2a+HqB0nmAcc3c2wH7FlVXwVOBxYA84BHgJ0m83ySJEmSJEmaGbPZgTQ3yWpge9pb+S4Gzhoj/heB5Ul+AjwJ/H5V/aC59pkkzwF+ApxUVQ+Pt3hVPZzkfNpnbn0PuKm5NAD8TZIFQIC/bGK/AFye5FeBPxh+bpYkSZIkSZJmX0ZpjFKHWq1WDQ4OdjsNSZIkSZKkbUaSVVXVGulaTx8AL0mSJEmSJA3V9YPOkxxF++2EQ91TVcdNcJ6PAocNGz6nqi6cSn6SJEmSJEnqHV0vZlXVtcC10zDPSdOQjiRJkiRJknqY2wwlSZIkSZLUNyxmSZIkSZIkqW9YzJIkSZIkSVLfsJglSZIkSZKkvmExS5IkSZIkSX2j628z7HdrN2xk0fKru52GJElSz1q/Ymm3U5AkSduQnujMSrIlyeoka5LcnOTQDu6Zn+S+JOeOE7c+yS6TyOmEJD8/0fskSZIkSZI0c3qimAVsrqolVXUg8F7gzzu4538C/ziDOZ0AWMySJEmSJEnqIb1SzBpqPvDQWAFJXgY8F/jyRCZO8rkkq5KsS7KsGRtIclGS25KsTfLuJMcDLeCSpmNs7iSfRZIkSZIkSdOoV87MmptkNTAH2A04crTAJNsB/xt4K/DaCa7z21X1YFOcuinJZ4BFwO5VtV8z/85V9XCSk4HTqmpwwk8jSZIkSZKkGdErnVlbtxkuBo4GPpkko8T+PvDFqrpvEuuckmQN8HVgT2Af4G5g7yQfSXI08MPxJkmyLMlgksEtj26cRBqSJEmSJEmajF7pzHpKVX2tObB9IXD/CCGvAA5P8vvAPODZSTZV1fKx5k1yBO1OrldU1aNJrgPmVNVDSQ4EjgJOBN4E/PY4Oa4EVgLssNs+NYHHkyRJkiRJ0hT0XDEryWJgAHhgpOtV9ZtDYk8AWuMVshoLgIeaQtZi4OXNHLsAP66qzyT5JvA3TfwjwE6TfhBJkiRJkiRNu14pZm09MwsgwNurass0r3ENcGKSO4Bv0t5qCLA7cGFzFhe036YIcBFwXpLNtLu5Nk9zPpIkSZIkSZqgVLlLbiparVYNDnpGvCRJkiRJ0nRJsqqqWiNd65UD4CVJkiRJkqRx9co2w6dJsj9w8bDhx6vqkFHibwR2GDb8tqpaOxP5SZIkSZIkafb1bDGrKUItmUD8iEUuSZIkSZIkbTvcZihJkiRJkqS+YTFLkiRJkiRJfcNiliRJkiRJkvqGxSxJkiRJkiT1DYtZkiRJkiRJ6hs9+zbDfrF2w0YWLb+622lIkrRNW79iabdTkCRJUo+wM0uSJEmSJEl9oyeKWUm2JFmdZE2Sm5McOk78NUkeTnJVB3Nfl6Q1iZyOTfLiid4nSZIkSZKkmdMTxSxgc1UtqaoDgfcCfz5O/IeAt81wTscCFrMkSZIkSZJ6SK8Us4aaDzw0VkBVfQV4ZKITJ/l4ksEk65J8YMj4iiS3J7k1yYebzrA3AB9qOsZeMNG1JEmSJEmSNP165QD4uUlWA3OA3YAjZ2id91XVg0kGgK8kOQDYABwHLK6qSrJzVT2c5Ergqqq6fPgkSZYBywAG5i+coVQlSZIkSZI0XK90Zm3dZrgYOBr4ZJLMwDpvSnIzcAuwL+1thBuBx4ALkrwReHS8SapqZVW1qqo1sOOCGUhTkiRJkiRJI+mVYtZTquprwC7AtLY8JdkLOA14TVUdAFwNzKmqJ4CDgcuBY4BrpnNdSZIkSZIkTZ9e2Wb4lCSLgQHggWmeej7wI2BjkucCrweuSzIP2LGqvpjkBuDuJv4RYKdpzkGSJEmSJElT0CvFrK1nZgEEeHtVbRktOMn1wGJgXpL7gN+pqmvHWqCq1iS5BbgTuBe4obm0E/D5JHOatd/TjH8KOD/JKcDxVXXXSPPuv/sCBlcs7eQZJUmSJEmSNEU9UcyqqoEJxh8+gdgjhnw+YZSwg0e47wbaZ2pJkiRJkiSpR/TcmVmSJEmSJEnSaHqiM2skSfYHLh42/HhVHTJK/BXAXsOGTx9v+6EkSZIkSZL6R88Ws6pqLbBkAvHHzVw2kiRJkiRJ6gVuM5QkSZIkSVLfsJglSZIkSZKkvmExS5IkSZIkSX3DYpYkSZIkSZL6Rs8eAN8v1m7YyKLlV3c7DUmSZtX6FUu7nYIkSZKeoezMkiRJkiRJUt/omWJWkvclWZfk1iSrkxwyStwFSdY0cZcnmTfbuUqSJEmSJKk7eqKYleQVwDHAS6vqAOC1wL2jhL+7qg5s4v4VOHmW0pQkSZIkSVKX9UQxC9gN+EFVPQ5QVT+oqn8bKbCqfgiQJMBcoEabNMmvJLkxyS1J/m+S5ybZLsn6JDsPiftWc+0FSb6eZG2SP02yaTofUpIkSZIkSVPTK8WsLwN7JvmXJB9L8qqxgpNcCHwPWAx8ZIzQfwJeXlUvAT4F/LeqehL4PHBcM9chwHeq6vvAOcA5VbU/cN8Y6y9LMphkcMujGzt/SkmSJEmSJE1JTxSzqmoT8DJgGfDvwGVJThgj/h3AzwN3AG8eY+o9gGuTrAX+ENi3Gb9syH1vab4DvAL4dPP50jHWX1lVrapqDey4YIzlJUmSJEmSNJ16opgFUFVbquq6qno/7XOwfm28eNrdVmPFfQQ4t+m0+i/AnGb8a8ALkywEjgU+O8X0JUmSJEmSNAt6opiV5EVJ9hkytAT4zghxSfLCrZ+BNwB3jjH1AmBD8/ntWwerqoArgLOAO6rqgebS1/mP4thbJv4kkiRJkiRJmknP6nYCjXnAR5pD2Z8Avk17y+FwAf46yfzm8xrgv44x7xnAp5M8BPw/YK8h1y4DbgJOGDL2LuBvkrwPuAYY90Cs/XdfwOCKpeOFSZIkSZIkaRr0RDGrqlYBh3YQ9yRw2ATm/Tztw95HujZIuyA21AbaB8ZXkrcAL+p0LUmSJEmSJM28nihm9ZCXAec2WxgfBn67u+lIkiRJkiRpqJ4tZiW5gp/eFghwelVdO0Ls+4BfHzb86ar6s4msWVXXAwdOKFFJkiRJkiTNmp4tZlXVcROI/TNgQoUrSZIkSZIk9Z+eeJuhJEmSJEmS1AmLWZIkSZIkSeobFrMkSZIkSZLUNyxmSZIkSZIkqW/07AHw/WLtho0sWn51t9OQJPWY9SuWdjsFSZIkaZtkZ5YkSZIkSZL6Rk8Us5JsSbI6yZokNyc5dIzY5zcxq5OsS3LiOHOvT7LLJHI6IcnPT/Q+SZIkSZIkzZxe2Wa4uaqWACQ5Cvhz4FWjxH4XeEVVPZ5kHnBbkiur6t+mOacTgNuA6Z5XkiRJkiRJk9Qrxayh5gMPjXaxqn485OsOTKC7LMnngD2BOcA5VbUyyQBwAdACCvgr4N7m+yVJNtMunm2e4HNIkiRJkiRpmvVKMWtuktW0i0y7AUeOFZxkT+Bq4IXAH06gK+u3q+rBJHOBm5J8BlgE7F5V+zVz71xVDyc5GTitqgZHWH8ZsAxgYP7CDpeWJEmSJEnSVPXEmVk02wyrajFwNPDJJBktuKruraoDaBez3p7kuR2uc0qSNcDXaXdo7QPcDeyd5CNJjgZ+ON4kVbWyqlpV1RrYcUGHS0uSJEmSJGmqeqWY9ZSq+hqwCzBuy1PTkXUbcPh4sUmOAF5Le8vggcAtwJyqegg4ELgOOBH4xCRTlyRJkiRJ0gzruWJWksXAAPDAKNf3aLYJkuRngV8EvtnB1AuAh6rq0WaNlzdz7AJsV1WfAf478NIm/hFgp6k8iyRJkiRJkqZXr52ZBRDg7VW1ZZTYXwD+d5JqYj9cVWs7WOMa4MQkd9Aufn29Gd8duDDJ1sLee5u/LwLOG+8A+P13X8DgiqUdLC9JkiRJkqSpSlV1O4e+1mq1anDwaWfES5IkSZIkaZKSrKqq1kjXem6boSRJkiRJkjSaXtlm+DRJ9gcuHjb8eFUdMkr8jcAOw4bf1uEWREmSJEmSJPWBni1mNUWoJROIH7HIJUmSJEmSpG2H2wwlSZIkSZLUNyxmSZIkSZIkqW9YzJIkSZIkSVLfsJglSZIkSZKkvtGzB8D3i7UbNrJo+dXdTkOSnjHWr1ja7RQkSZIkdZGdWZIkSZIkSeobs1LMSrIlyeok65KsSXJqknHXTvK8JJuSnNZ83zPJV5Pc3sz1zpnPXpIkSZIkSb1itrYZbq6qJQBJdgUuBeYD7x/nvrOALw35/gRwalXdnGQnYFWSv6+q22cgZ0mSJEmSJPWYWd9mWFX3A8uAk5NktLgkxwL3AOuG3Pvdqrq5+fwIcAew+xhz/F6Sm5pusM8k2THJgiTf2doZluRnktybZPskByW5teki+1CS26bloSVJkiRJkjQtunJmVlXdDQwAu450Pck84HTgA6PNkWQR8BLgxjGW+mxVHVRVB9IufP1OVW0EVgOvamKOAa6tqp8AFwL/peki2zLG2suSDCYZ3PLoxjGWlyRJkiRJ0nTq1QPgzwDOrqpNI11sil2fAd5VVT8cY579klyfZC3wm8C+zfhlwJubz28BLkuyM7BTVX2tGb90tEmramVVtaqqNbDjgk6fSZIkSZIkSVM0W2dm/ZQke9PufLp/lJBDgOOTfBDYGXgyyWNVdW6S7WkXsi6pqs+Os9RFwLFVtSbJCcARzfiVwP9K8nPAy4D/B+w0+SeSJEmSJEnSbJj1YlaShcB5wLlVVSPFVNXhQ+LPADY1hawAFwB3VNVZHSy3E/DdpgD2m8CGZv5NSW4CzgGuqqotwMNJHklySFXdSLtjS5IkSZIkST1ktopZc5OsBran/UbCi2m/qXCiDgPeBqxt5gP4o6r64ijx/4P2mVr/3vw9tPvqMuDT/Ee3FsDvAOcneRL4B8ADsSRJkiRJknpIRmmOekZKMm/rOV1JlgO7VdU7x7qn1WrV4ODgrOQnSZIkSZL0TJBkVVW1RrrWlTOzetjSJO+l/d/lO8AJ3U1HkiRJkiRJQ3W1mJXkKODMYcP3VNVxE5zno7S3IA51TlVdOJF5quoy2tsPJUmSJEmS1IO6WsyqqmuBa6dhnpOmIR1JkiRJkiT1uO26nYAkSZIkSZLUKYtZkiRJkiRJ6hsWsyRJkiRJktQ3LGZJkiRJkiSpb1jMkiRJkiRJUt/o6tsMtwVrN2xk0fKru52GJHXN+hVLu52CJEmSpGeQnunMSvK+JOuS3JpkdZJDRom7IMmaJu7yJPPGmPOMJKdNIpclSX55ovdJkiRJkiRpZvVEMSvJK4BjgJdW1QHAa4F7Rwl/d1Ud2MT9K3DyDKS0BLCYJUmSJEmS1GN6opgF7Ab8oKoeB6iqH1TVv40UWFU/BEgSYC5QnSyQ5PeS3NR0dX0myY7N+K8nua0Z/8ckzwb+BHhz0yH25ml4PkmSJEmSJE2DXilmfRnYM8m/JPlYkleNFZzkQuB7wGLgIx2u8dmqOqiqDgTuAH6nGf9j4Khm/A1V9eNm7LKqWlJVl42w/rIkg0kGtzy6scPlJUmSJEmSNFU9Ucyqqk3Ay4BlwL8DlyU5YYz4dwA/T7so1Wnn1H5Jrk+yFvhNYN9m/AbgoiS/Bwx0mO/KqmpVVWtgxwUdLi9JkiRJkqSp6oliFkBVbamq66rq/bTPwfq18eKBT40XN8RFwMlVtT/wAWBOM8+JwH8H9gRWJXnO5J5AkiRJkiRJM60nillJXpRknyFDS4DvjBCXJC/c+hl4A3Bnh8vsBHw3yfa0O7O2zvmCqrqxqv6YdlfYnsAjTbwkSZIkSZJ6yLO6nUBjHvCRJDsDTwDfpr3lcLgAf51kfvN5DfBfO1zjfwA30i5Y3ch/FKs+1BTSAnylmfNfgeVJVgN/PtK5WZIkSZIkSZp9qeroZYAaRavVqsHBwW6nIUmSJEmStM1IsqqqWiNd64lthpIkSZIkSVInemWb4dMkuQLYa9jw6VV17Qix7wN+fdjwp6vqz2YqP0mSJEmSJM2+ni1mVdVxE4j9M8DClSRJkiRJ0jbObYaSJEmSJEnqGxazJEmSJEmS1DcsZkmSJEmSJKlvWMySJEmSJElS37CYJUmSJEmSpL7Rs28z7BdrN2xk0fKru52GJE3Z+hVLu52CJEmSJI1r1jqzkmxJsjrJuiRrkpyaZNz1kzwvyaYkpw0Z+6sk9ye5rYP7L0py/CTyPSLJoRO9T5IkSZIkSTNnNrcZbq6qJVW1L/A64PXA+zu47yzgS8PGLgKOnt70nuYIwGKWJEmSJElSD+nKmVlVdT+wDDg5SUaLS3IscA+wbtj9/wg8ONF1k/xxkpuS3JZk5da1k5yS5PYktyb5VJJFwInAu5tussMnupYkSZIkSZKmX9cOgK+qu4EBYNeRrieZB5wOfGAalz23qg6qqv2AucAxzfhy4CVVdQBwYlWtB84Dzm66ya4fltuyJINJBrc8unEa05MkSZIkSdJYevlthmfQLiZtmsY5X53kxiRrgSOBfZvxW4FLkrwVeGK8SapqZVW1qqo1sOOCaUxPkiRJkiRJY+na2wyT7A1sAe4fJeQQ4PgkHwR2Bp5M8lhVnTvJ9eYAHwNaVXVvkjOAOc3lpcArgV8B3pdk/8msIUmSJEmSpJnVlWJWkoW0t/GdW1U1UkxVHT4k/gxg02QLWY2thasfNFsYjwcub96ouGdVfTXJPwFvAeYBjwDzp7CeJEmSJEmSptlsbjOc2xymvg74v8CXmeR5WEn+Fvga8KIk9yX5nfHuqaqHgfOB24BrgZuaSwPA3zRbD28B/rKJ/QJwnAfAS5IkSZIk9Y6M0hilDrVarRocHOx2GpIkSZIkSduMJKuqqjXStV4+AF6SJEmSJEn6KV07AH6rJEcBZw4bvqeqjpvgPB8FDhs2fE5VXTiV/CRJkiRJktQ7ul7MqqpraZ9hNdV5TpqGdCRJkiRJktTD3GYoSZIkSZKkvmExS5IkSZIkSX3DYpYkSZIkSZL6hsUsSZIkSZIk9Q2LWZIkSZIkSeobXX+bYb9bu2Eji5Zf3e00JGnK1q9Y2u0UJEmSJGlcs9aZlWRLktVJ1iVZk+TUJOOun+R5STYlOa35PifJN5o51iX5wDj3X5ekNYl8j03y4oneJ0mSJEmSpJkzm9sMN1fVkqraF3gd8Hrg/R3cdxbwpSHfHweOrKoDgSXA0UlePt3JAscCFrMkSZIkSZJ6SFfOzKqq+4FlwMlJMlpckmOBe4B1Q+6tqtrUfN2++VOdrJvk40kGh3d0JVmR5PYktyb5cJJDgTcAH2q6yV4w0WeUJEmSJEnS9OvamVlVdXeSAWBX4PvDryeZB5xOu4vrtGHXBoBVwAuBj1bVjR0u+76qerC5/ytJDgA2AMcBi6uqkuxcVQ8nuRK4qqouHyG3ZbSLcQzMX9jh0pIkSZIkSZqqXn6b4RnA2UO6sJ5SVVuqagmwB3Bwkv06nPNNSW4GbgH2pb2NcCPwGHBBkjcCj443SVWtrKpWVbUGdlzQ4dKSJEmSJEmaqq51ZiXZG9gC3D9KyCHA8Uk+COwMPJnksao6d2tA00H1VeBo4LZx1tuLdofXQVX1UJKLgDlV9USSg4HXAMcDJwNHTunhJEmSJEmSNCO6UsxKshA4Dzi3qkY876qqDh8SfwawqarObe79SVPImkt7G+KZHSw7H/gRsDHJc2kfQH9ds51xx6r6YpIbgLub+EeAnSb3hJIkSZIkSZoJs1nMmptkNe0D258ALqb9psKJ2g346+bcq+2Av6uqq8a7qarWJLkFuBO4F7ihubQT8Pkkc4AA72nGPwWcn+QU4PiqumsSuUqSJEmSJGkaZZTGKHWo1WrV4OBgt9OQJEmSJEnaZiRZVVWtka718gHwkiRJkiRJ0k/p2gHwWyU5iqefeXVPVR03wXmuAPYaNnx6VV07lfwkSZIkSZLUO7pezGqKTVMuOE20+CVJkiRJkqT+4zZDSZIkSZIk9Q2LWZIkSZIkSeobFrMkSZIkSZLUNyxmSZIkSZIkqW9YzJIkSZIkSVLf6PrbDPvd2g0bWbT86m6nIUlPWb9iabdTkCRJkqQZM2udWUm2JFmdZF2SNUlOTTLu+kmel2RTktOa73OSfKOZY12SD8x89pIkSZIkSeoFs9mZtbmqlgAk2RW4FJgPvH+c+84CvjTk++PAkVW1Kcn2wD8l+VJVfX0GcpYkSZIkSVIP6cqZWVV1P7AMODlJRotLcixwD7BuyL1VVZuar9s3f2qMOf44yU1JbkuyMm2Lk3xjSMyiJGubz7+c5M4kq5L8ZZKrpvKskiRJkiRJmj5dOwC+qu4GBoBdR7qeZB5wOvC0bYRJBpKsBu4H/r6qbhxjqXOr6qCq2g+YCxxTVXcCz06yVxPzZuCyJHOA/wO8vqpeBiwcJbdlSQaTDG55dGMnjytJkiRJkqRp0MtvMzwDOHtIF9ZTqmpLs2VxD+DgJPuNMc+rk9zYdF4dCezbjP8d7SIWzd+XAYuBu6vqnmb8b0easKpWVlWrqloDOy6Y4GNJkiRJkiRpsrr2NsMkewNbaHdXjeQQ4PgkHwR2Bp5M8lhVnbs1oKoeTvJV4GjgthHWmAN8DGhV1b1JzgDmNJcvAz6d5LPtqepbSZZMy8NJkiRJkiRpRnSlMyvJQuA82lsARzzvqqoOr6pFVbUI+Avgf1XVuUkWJtm5mWcu8DrgzlGW2lq4+kGzbfH4IfPfRbuY9j9oF7YAvgnsnWRR831r55YkSZIkSZJ6wGx2Zs1tzrnaHngCuJj2mwonajfgr5MM0C7G/V1VjXhIe9O5dT7trq3vATcNC7kM+BCwVxO/OcnvA9ck+dEI8ZIkSZIkSeqijNIY9YyVZF5VbWresvhR4FtVdfZo8a1WqwYHB2cvQUmSJEmSpG1cklVV1RrpWi8fAN8tv9d0kK0DFtB+u6EkSZIkSZJ6QNcOgN8qyVHAmcOG76mq4yY4zxU02wWHOL2qrp3IPE0X1qidWJIkSZIkSeqerhezmmLThApOo8wzoeKXJEmSJEmS+o/bDCVJkiRJktQ3LGZJkiRJkiSpb1jMkiRJkiRJUt+wmCVJkiRJkqS+YTFLkiRJkiRJfaPrbzPsd2s3bGTR8qu7nYYkPWX9iqXdTkGSJEmSZsysdWYl2ZJkdZJ1SdYkOTXJuOsneV6STUlOGzL2ziS3NXO9a5z7L0py/CTyPSLJoRO9T5IkSZIkSTNnNjuzNlfVEoAkuwKXAvOB949z31nAl7Z+SbIf8HvAwcCPgWuSXFVV357mfI8ANgH/PM3zSpIkSZIkaZK6cmZWVd0PLANOTpLR4pIcC9wDrBsy/AvAjVX1aFU9AfwD8MZO1k3yx0luarq6Vm5dO8kpSW5PcmuSTyVZBJwIvLvpJjt8Ms8pSZIkSZKk6dW1A+Cr6m5gANh1pOtJ5gGnAx8Yduk24PAkz0myI/DLwJ4dLntuVR1UVfsBc4FjmvHlwEuq6gDgxKpaD5wHnF1VS6rq+gk8miRJkiRJkmZIL7/N8AzaxaRNQwer6g7gTODLwDXAamBLh3O+OsmNSdYCRwL7NuO3ApckeSvwxHiTJFmWZDDJ4JZHN3a4tCRJkiRJkqaqa8WsJHvTLkLdP0rIIcAHk6wH3gX8UZKTAarqgqp6WVW9EngI+JcO1psDfAw4vqr2B84H5jSXlwIfBV4K3JRkzLPEqmplVbWqqjWw44LxlpYkSZIkSdI0mc0D4J+SZCHtbXznVlWNFFNVhw+JPwPYVFXnNt93rar7kzyP9nlZL+9g2a2Fqx80WxiPBy5v3qi4Z1V9Nck/AW8B5gGP0D6gXpIkSZIkST1iNotZc5OsBranvZXvYtpvKpyMzyR5DvAT4KSqeni8G6rq4STn0z5z63vATc2lAeBvkiwAAvxlE/sF2sWuXwX+wHOzJEmSJEmSui+jNEapQ61WqwYHB7udhiRJkiRJ0jYjyaqqao10rZcPgJckSZIkSZJ+SlfOzBoqyVG030441D1VddwE5/kocNiw4XOq6sKp5CdJkiRJkqTe0fViVlVdC1w7DfOcNA3pSJIkSZIkqYe5zVCSJEmSJEl9w2KWJEmSJEmS+obFLEmSJEmSJPUNi1mSJEmSJEnqGxazJEmSJEmS1De6/jbDfrd2w0YWLb+622lI6mHrVyztdgqSJEmStM2wM0uSJEmSJEl9oyeKWUm2JFmdZE2Sm5McOkbskiRfS7Iuya1J3jzO3NclaU0ip2OTvHii90mSJEmSJGnm9EQxC9hcVUuq6kDgvcCfjxH7KPBbVbUvcDTwF0l2noGcjgUsZkmSJEmSJPWQXilmDTUfeGi0i1X1L1X1rebzvwH3Aws7mTjJx5MMNl1dHxgyviLJ7U2n14ebzrA3AB9qOsZeMKUnkiRJkiRJ0rTolQPg5yZZDcwBdgOO7OSmJAcDzwbu6nCd91XVg0kGgK8kOQDYABwHLK6qSrJzVT2c5Ergqqq6fIR1lwHLAAbmd1RHkyRJkiRJ0jTolc6srdsMF9PeOvjJJBnrhiS7ARcD76iqJztc501JbgZuAfalvY1wI/AYcEGSN9LexjimqlpZVa2qag3suKDDpSVJkiRJkjRVvVLMekpVfQ3YhTG2DiaZD1xNu9Pq653Mm2Qv4DTgNVV1QHP/nKp6AjgYuBw4Brhmak8gSZIkSZKkmdIr2wyfkmQxMAA8MMr1ZwNXAJ8caQvgGOYDPwI2Jnku8HrguiTzgB2r6otJbgDubuIfAXaa5GNIkiRJkiRpBvRKMWvrmVkAAd5eVVtGiX0T8ErgOUlOaMZOqKrVo8QDUFVrktwC3AncC9zQXNoJ+HySOc3a72nGPwWcn+QU4PiqGvFcrv13X8DgiqXjPJ4kSZIkSZKmQ6qq2zn0tVarVYODg91OQ5IkSZIkaZuRZFVVtUa61nNnZkmSJEmSJEmj6ZVthk+TZH/abysc6vGqOmSU+CuAvYYNn15V185EfpIkSZIkSZp9PVvMqqq1wJIJxB83c9lIkiRJkiSpF7jNUJIkSZIkSX3DYpYkSZIkSZL6hsUsSZIkSZIk9Q2LWZIkSZIkSeobPXsAfL9Yu2Eji5Zf3e00JPWw9SuWdjsFSZIkSdpm2JklSZIkSZKkvjFrxawkW5KsTrIuyZokpyYZd/0kz0uyKclpQ8Z2TnJ5kjuT3JHkFWPcf1GS4yeR7xFJDp3ofZIkSZIkSZo5s7nNcHNVLQFIsitwKTAfeP84950FfGnY2DnANVV1fJJnAztOc64ARwCbgH+egbklSZIkSZI0CV3ZZlhV9wPLgJOTZLS4JMcC9wDrhowtAF4JXNDM9eOqeriTdZP8cZKbktyWZOXWtZOckuT2JLcm+VSSRcCJwLubbrLDJ/WgkiRJkiRJmlZdOzOrqu4GBoBdR7qeZB5wOvCBYZf2Av4duDDJLUk+keRnOlz23Ko6qKr2A+YCxzTjy4GXVNUBwIlVtR44Dzi7qpZU1fXDcluWZDDJ4JZHN3a4tCRJkiRJkqaqlw+AP4N2MWnTsPFnAS8FPl5VLwF+RLsY1YlXJ7kxyVrgSGDfZvxW4JIkbwWeGG+SqlpZVa2qag3suKDDpSVJkiRJkjRVs3lm1k9JsjewBbh/lJBDgOOTfBDYGXgyyWPA5cB9VXVjE3c5HRSzkswBPga0qureJGcAc5rLS2lvXfwV4H1J9p/UQ0mSJEmSJGlGdaWYlWQh7W1851ZVjRRTVYcPiT8D2FRV5zbf703yoqr6JvAa4PYOlt1auPpBs4XxeODy5o2Ke1bVV5P8E/AWYB7wCO0D6iVJkiRJktQjZrOYNTfJamB72lv5Lqb9psLJ+APa2wKfDdwNvGO8G6rq4STnA7cB3wNuai4NAH/THCwf4C+b2C/QLnb9KvAHw8/N2mr/3RcwuGLpJB9DkiRJkiRJE5FRGqPUoVarVYODg91OQ5IkSZIkaZuRZFVVtUa61ssHwEuSJEmSJEk/pWsHwG+V5CjgzGHD91TVcROc56PAYcOGz6mqC6eSnyRJkiRJknpH14tZVXUtcO00zHPSNKQjSZIkSZKkHuY2Q0mSJEmSJPUNi1mSJEmSJEnqGxazJEmSJEmS1DcsZkmSJEmSJKlvdP0A+H63dsNGFi2/uttpSJoF61cs7XYKkiRJkvSMZ2eWJEmSJEmS+kZPFLOSbEmyOsltSb6QZOdx4q9J8nCSqzqY+7okrUnkdGySF0/0PkmSJEmSJM2cnihmAZuraklV7Qc8CJw0TvyHgLfNcE7HAhazJEmSJEmSekivFLOG+hqw+1gBVfUV4JGJTpzk40kGk6xL8oEh4yuS3J7k1iQfTnIo8AbgQ03H2AsmupYkSZIkSZKmX08dAJ9kAHgNcMEMLfG+qnqwWecrSQ4ANgDHAYurqpLsXFUPJ7kSuKqqLh8hz2XAMoCB+QtnKFVJkiRJkiQN1yudWXOTrAa+BzwX+PsZWudNSW4GbgH2pb2NcCPwGHBBkjcCj443SVWtrKpWVbUGdlwwQ6lKkiRJkiRpuF4pZm2uqiXA84Ew/plZE5ZkL+A04DVVdQBwNTCnqp4ADgYuB44BrpnutSVJkiRJkjQ9eqWYBUBVPQqcApyaZLq3QM4HfgRsTPJc4PUASeYBC6rqi8C7gQOb+EeAnaY5B0mSJEmSJE3BmAWjJO8Z63pVnTW96UBV3ZLkVuA3gItHyet6YDEwL8l9wO9U1bXjzLsmyS3AncC9wA3NpZ2AzyeZQ7srbOszfwo4P8kpwPFVdddI8+6/+wIGVyyd0DNKkiRJkiRpcsbrfpqVzqSqmjfs+6+ME3/4BOY+YsjnE0YJO3iE+26gfaaWJEmSJEmSesSYxayq+sBsJSJJkiRJkiSNp6NzqZLsAXwEOKwZuh54Z1XdN1OJJdmfp28zfLyqDhkl/gpgr2HDp4+3/VCSJEmSJEn9o9ND1i8ELgV+vfn+1mbsdTORFEBVrQWWTCD+uJnKRZIkSZIkSb2h07cZLqyqC6vqiebPRcDCGcxLkiRJkiRJeppOi1kPJHlrkoHmz1uBB2YyMUmSJEmSJGm4TotZvw28Cfge8F3geOCEGcpJkiRJkiRJGlGnZ2b9CfD2qnoIIMnPAR+mXeSSJEmSJEmSZkWnnVkHbC1kAVTVg8BLZiYlSZIkSZIkaWSddmZtl+Rnh3VmdXrvNm3tho0sWn51t9OQnpHWr1ja7RQkSZIkSbOs086s/w18Lcn/TPI/gX8GPjiRhZJsSbI6yZokNyc5dJz4a5I8nOSqYeMnJ/l2kkqyyzhznJDk3Ink2dy3KMl/nuh9kiRJkiRJmlkdFbOq6pPAG4HvN3/eWFUXT3CtzVW1pKoOBN4L/Pk48R8C3jbC+A3Aa4HvTHD9iVgEWMySJEmSJEnqMR1vFayq24Hbp2nd+cBDYwVU1VeSHDHC+C0ASSa0YJJfAf478GzgAeA3q+r7SV4FnLN1euCVwArgF5KsBv66qs6e0GKSJEmSJEmaEbN57tXcpjg0B9gNOHIW1wb4J+DlVVVJfhf4b8CpwGnASVV1Q5J5wGPAcuC0qjpmpImSLAOWAQzMXzgryUuSJEmSJGl2i1mbq2oJQJJXAJ9Msl9V1SytvwdwWZLdaHdn3dOM3wCcleQS4LNVdd94XV9VtRJYCbDDbvvMVv6SJEmSJEnPeJ0eAD+tquprwC7AbLY1fQQ4t6r2B/4L7Q4xqmoF8LvAXOCGJItnMSdJkiRJkiRNwGx2Zj2lKRgN0D67arYsADY0n98+JJcXVNVaYG2Sg4DFwL3ATrOYmyRJkiRJkjowm51Zc5Osbs7Nugx4e1VtGS04yfXAp4HXJLkvyVHN+ClJ7qO9bfDWJJ/ocP0zgE8nWQX8YMj4u5LcluRW4CfAl4BbgS1J1iR598QeU5IkSZIkSTMls3dk1bap1WrV4OBgt9OQJEmSJEnaZiRZVVWtka515cwsSZIkSZIkaTK6cmbWVkn2By4eNvx4VR0ywXneAbxz2PANVXXSVPKTJEmSJElSb+lqMas5eH3JNMxzIXDhlBOSJEmSJElST3OboSRJkiRJkvqGxSxJkiRJkiT1DYtZkiRJkiRJ6hsWsyRJkiRJktQ3LGZJkiRJkiSpb3T1bYbbgrUbNrJo+dXdTkN6Rlq/Ymm3U5AkSZIkzbJZ6cxKsiXJ6iTrkqxJcmqScddO8rwkm5Kc1nzfM8lXk9zezPXOacrvn6djHkmSJEmSJM2s2erM2lxVSwCS7ApcCswH3j/OfWcBXxry/Qng1Kq6OclOwKokf19Vt08luao6dCr3S5IkSZIkaXbM+plZVXU/sAw4OUlGi0tyLHAPsG7Ivd+tqpubz48AdwC7jzHHdUnOTjKY5I4kByX5bJJvJfnTIXGbmr+PaO65PMmdSS4ZK0dJkiRJkiTNrq4cAF9VdwMDwK4jXU8yDzgd+MBocyRZBLwEuHGc5X5cVS3gPODzwEnAfsAJSZ4zQvxLgHcBLwb2Bg4bYe1lTYFscMujG8dZXpIkSZIkSdOlV99meAZwdlVtGuliU+z6DPCuqvrhOHNd2fy9FljXdHc9DtwN7DlC/Deq6r6qehJYDSwaHlBVK6uqVVWtgR0XdPI8kiRJkiRJmgZdeZthkr2BLcD9o4QcAhyf5IPAzsCTSR6rqnOTbE+7kHVJVX22g+Ueb/5+csjnrd9Hev6hMVtGiZEkSZIkSVIXzHqhJslC2lv+zq2qGimmqg4fEn8GsKkpZAW4ALijqs6ajXwlSZIkSZLUO2armDU3yWpge9pvJLyY9psKJ+ow4G3A2mY+gD+qqi9OR5KSJEmSJEnqbRmlOUodarVaNTg42O00JEmSJEmSthlJVjUv9HuaXj0AXpIkSZIkSXqarh5unuQo4Mxhw/dU1XETnOejtLcgDnVOVV04lfwkSZIkSZLUW7pazKqqa4Frp2Gek6YhHUmSJEmSJPU4txlKkiRJkiSpb1jMkiRJkiRJUt+wmCVJkiRJkqS+YTFLkiRJkiRJfcNiliRJkiRJkvpGV99muC1Yu2Eji5Zf3e00pJ6wfsXSbqcgSZIkSdrG9URnVpItSVYnWZPk5iSHdhi/OsmV48Rel6Q1iZyOTfLiid4nSZIkSZKkmdMrnVmbq2oJQJKjgD8HXtVJ/Aw6FrgKuH2G15EkSZIkSVKHeqIza5j5wEMzMXGSjycZTLIuyQeGjK9IcnuSW5N8uOkMewPwoab76wUzkY8kSZIkSZImplc6s+YmWQ3MAXYDjhwnfk6SQeAJYEVVfa7Ddd5XVQ8mGQC+kuQAYANwHLC4qirJzlX1cLN98aqqunz4JEmWAcsABuYv7HBpSZIkSZIkTVWvdGZtrqolVbUYOBr4ZJKMEf/8qmoB/xn4iwl0Tr0pyc3ALcC+wIuBjcBjwAVJ3gg8Ot4kVbWyqlpV1RrYcUGHS0uSJEmSJGmqeqWY9ZSq+hqwCzBqy1NVbWj+vhu4DnjJePMm2Qs4DXhNVR0AXA3MqaongIOBy4FjgGum+AiSJEmSJEmaIT1XzEqyGBgAHhjl+s8m2aH5vAtwGJ0d0j4f+BGwMclzgdc3c8wDFlTVF4F3Awc28Y8AO03hUSRJkiRJkjTNeu3MLIAAb6+qLaPE/gLwf5I8SbsYt6Kqxi1mVdWaJLcAdwL3Ajc0l3YCPp9kTrP2e5rxTwHnJzkFOL6q7prEc0mSJEmSJGkapaq6nUNfa7VaNTg42O00JEmSJEmSthlJVjXnpT9Nz20zlCRJkiRJkkbTK9sMnybJ/sDFw4Yfr6pDRom/Athr2PDpVXXtTOQnSZIkSZKk2dezxayqWgssmUD8cTOXjSRJkiRJknqB2wwlSZIkSZLUNyxmSZIkSZIkqW9YzJIkSZIkSVLfsJglSZIkSZKkvmExS5IkSZIkSX2jZ99m2C/WbtjIouVXdzsNqSesX7G02ylIkiRJkrZxM9aZlWRLktVJ1iVZk+TUJKOul+TgJn51E3/ckGvvTHJbM9e7ZipnSZIkSZIk9baZ7MzaXFVLAJLsClwKzAfeP0r8bUCrqp5IshuwJskXgMXA7wEHAz8GrklyVVV9ewZzlyRJkiRJUg+alTOzqup+YBlwcpKMEvNoVT3RfJ0DVPP5F4Abh1z/B+CNo62V5LokZycZTHJHkoOSfDbJt5L86ZC4zyVZ1XR7LWvGnt/E7ZJkuyTXJ/mlqf8XkCRJkiRJ0nSYtTOzquruJAPArsD3R4pJcgjwV8Dzgbc1XVq3AX+W5DnAZuCXgcFxlvtxVbWSvBP4PPAy4EHgriRnV9UDwG9X1YNJ5gI3JflMVX0nyZnAx4FvALdX1ZdHyHMZ7eIcA/MXTvQ/hSRJkiRJkiapp95mWFU3VtW+wEHAe5PMqao7gDOBLwPXAKuBLeNMdWXz91pgXVV9t6oeB+4G9myunZJkDfD1ZmyfJodP0N4OeSJw2ih5rqyqVlW1BnZcMLmHlSRJkiRJ0oTNWjEryd60i1D3jxfbFLA2Afs13y+oqpdV1SuBh4B/GWeKx5u/nxzyeev3ZyU5Angt8IqqOhC4hfbWRpLsCOzRxM8b98EkSZIkSZI0a2almJVkIXAecG5V1SgxeyV5VvP5+bQPfl/ffN+1+ft5tM/LunSKKS0AHqqqR5MsBl4+5NqZwCXAHwPnT3EdSZIkSZIkTaOZPDNrbpLVwPbAE8DFwFljxP8isDzJT2h3UP1+Vf2gufaZ5sysnwAnVdXDU8ztGuDEJHcA36S91ZAkr6K9xfGwqtqS5NeSvKOqLpziepIkSZIkSZoGGaVRSh1qtVo1ODjeefSSJEmSJEnqVJJVVdUa6VpPHQAvSZIkSZIkjWUmtxmOKMlRtM+lGuqeqjpugvN8FDhs2PA5bgmUJEmSJEnads16MauqrgWunYZ5TpqGdCRJkiRJktRH3GYoSZIkSZKkvmExS5IkSZIkSX3DYpYkSZIkSZL6hsUsSZIkSZIk9Q2LWZIkSZIkSeobs/42w23N2g0bWbT86m6nIfWE9SuWdjsFSZIkSdI2ric6s5JsSbI6yW1JvpBk5zFilyT5WpJ1SW5N8uZx5r4uSWsSOR2b5MUTvU+SJEmSJEkzpyeKWcDmqlpSVfsBDwInjRH7KPBbVbUvcDTwF2MVv6bgWMBiliRJkiRJUg/plWLWUF8Ddh/tYlX9S1V9q/n8b8D9wMJOJk7y8SSDTVfXB4aMr0hye9Pp9eEkhwJvAD7UdIy9YEpPJEmSJEmSpGnRU2dmJRkAXgNc0GH8wcCzgbs6XOJ9VfVgs85XkhwAbACOAxZXVSXZuaoeTnIlcFVVXT7xJ5EkSZIkSdJM6JXOrLlJVgPfA54L/P14NyTZDbgYeEdVPdnhOm9KcjNwC7Av7W2EG4HHgAuSvJH2Nsbx1l7WdHgNbnl0Y4dLS5IkSZIkaap6pZi1uaqWAM8HwthnZpFkPnA17U6rr3eyQJK9gNOA11TVAc39c6rqCeBg4HLgGOCa8eaqqpVV1aqq1sCOCzpZXpIkSZIkSdOgV4pZAFTVo8ApwKlJRtwCmeTZwBXAJye4BXA+8CNgY5LnAq9v5psHLKiqLwLvBg5s4h8BdprUg0iSJEmSJGlG9FQxC6CqbgFuBX5jlJA3Aa8ETmgOZ1+dZEkH866hvb3wTuBS4Ibm0k7AVUluBf4JeE8z/ingD5Pc4gHwkiRJkiRJvSFV1e0c+lqr1arBwcFupyFJkiRJkrTNSLKqqlojXeu5zixJkiRJkiRpNCOeS9ULkuxP+22FQz1eVYeMEn8FsNew4dOr6tqZyE+SJEmSJEmzr2eLWVW1FlgygfjjZi4bSZIkSZIk9QK3GUqSJEmSJKlvWMySJEmSJElS37CYJUmSJEmSpL5hMUuSJEmSJEl9w2KWJEmSJEmS+kbPvs2wX6zdsJFFy6/udhrStFq/Ymm3U5AkSZIkaUR2ZkmSJEmSJKlvzFgxK8mWJKuTrEuyJsmpScZdL8nzkmxKclrzfc8kX01yezPXO2cqZ0mSJEmSJPW2mdxmuLmqlgAk2RW4FJgPvH+c+84CvjTk+xPAqVV1c5KdgFVJ/r6qbp+BnCVJkiRJktTDZmWbYVXdDywDTk6S0eKSHAvcA6wbcu93q+rm5vMjwB3A7mPMcV2Ss5MMJrkjyUFJPpvkW0n+dEjc55Ksarq9ljVjz2/idkmyXZLrk/zSFB9fkiRJkiRJ02TWDoCvqruTDAC7At8ffj3JPOB04HXAaSPNkWQR8BLgxnGW+3FVtZotiZ8HXgY8CNyV5OyqegD47ap6MMlc4KYkn6mq7yQ5E/g48A3g9qr68gh5LKNdnGNg/sIOnl6SJEmSJEnToZcOgD8DOLuqNo10sSl2fQZ4V1X9cJy5rmz+Xgusa7q7HgfuBvZsrp2SZA3w9WZsH4Cq+gTt7ZAnMkpRrapWVlWrqloDOy7o9PkkSZIkSZI0RbPWmZVkb2ALcP8oIYcAxyf5ILAz8GSSx6rq3CTb0y5kXVJVn+1gucebv58c8nnr92clOQJ4LfCKqno0yXXAnCbPHYE9mvh5wCMdPaAkSZIkSZJm3KwUs5IsBM4Dzq2qGimmqg4fEn8GsKkpZAW4ALijqs6appQWAA81hazFwMuHXDsTuAT4DnA+cMw0rSlJkiRJkqQpmsli1twkq4Htab+R8GLabyqcqMOAtwFrm/kA/qiqvjiF3K4BTkxyB/BN2lsNSfIq4CDgsKrakuTXkryjqi4cbaL9d1/A4IqlU0hFkiRJkiRJncoojVLqUKvVqsHBwW6nIUmSJEmStM1IsqqqWiNd66UD4CVJkiRJkqQxzdoB8FslOYr2uVRD3VNVx01wno/S3oI41DljbQmUJEmSJElSf5v1YlZVXQtcOw3znDQN6UiSJEmSJKmPuM1QkiRJkiRJfcNiliRJkiRJkvqGxSxJkiRJkiT1DYtZkiRJkiRJ6huzfgD8tmbtho0sWn51t9OQRrR+xdJupyBJkiRJ0rSyM0uSJEmSJEl9oyeKWUm2JFmdZE2Sm5McOkbsq5vYrX8eS3LsGPHXJWlNIqdjk7x4ovdJkiRJkiRp5vTKNsPNVbUEIMlRwJ8DrxopsKq+CmyN/Tng28CXZyCnY4GrgNtnYG5JkiRJkiRNQk90Zg0zH3iow9jjgS9V1aOdBCf5eJLBJOuSfGDI+Ioktye5NcmHm86wNwAfarq/XjDhp5AkSZIkSdK065XOrLlJVgNzgN2AIzu87y3AWRNY531V9WCSAeArSQ4ANgDHAYurqpLsXFUPJ7kSuKqqLh8+SZJlwDKAgfkLJ7C8JEmSJEmSpqJXOrM2V9WSqloMHA18MknGuiHJbsD+wLUTWOdNSW4GbgH2BV4MbAQeAy5I8kZg3C6vqlpZVa2qag3suGACy0uSJEmSJGkqeqWY9ZSq+hqwCzBey9ObgCuq6iedzJtkL+A04DVVdQBwNTCnqp4ADgYuB44Brpls7pIkSZIkSZpZPVfMSrIYGAAeGCf0N4C/ncDU84EfARuTPBd4fbPePGBBVX0ReDdwYBP/CLDTBOaXJEmSJEnSDOu1M7MAAry9qraMFpxkEbAn8A+dLlBVa5LcAtwJ3Avc0FzaCfh8kjnN2u9pxj8FnJ/kFOD4qrprpHn3330BgyuWdpqGJEmSJEmSpqAnillVNTDB+PXA7h3GHjHk8wmjhB08wn030D5TS5IkSZIkST2i57YZSpIkSZIkSaPpic6skSTZH7h42PDjVXXIKPFXAHsNGz69qibytkNJkiRJkiT1sJ4tZlXVWmDJBOKPm7lsJEmSJEmS1AvcZihJkiRJkqS+YTFLkiRJkiRJfcNiliRJkiRJkvqGxSxJkiRJkiT1jZ49AL5frN2wkUXLr+52GtrGrV+xtNspSJIkSZLUE+zMkiRJkiRJUt+YtWJWki1JVidZl2RNklOTjLt+kucl2ZTktGHjA0luSXLVOPdfl6Q1iXyPTfLiid4nSZIkSZKkmTObnVmbq2pJVe0LvA54PfD+Du47C/jSCOPvBO6YxvyGOxawmCVJkiRJktRDurLNsKruB5YBJyfJaHFJjgXuAdYNG98DWAp8YiLrJvl4ksGmO+wDQ8ZXJLk9ya1JPpzkUOANwIeabrIXTGQdSZIkSZIkzYyuHQBfVXcnGQB2Bb4//HqSecDptLu4Tht2+S+A/wbsNMFl31dVDzbrfiXJAcAG4DhgcVVVkp2r6uEkVwJXVdXlI+S2jHYxjoH5CyeYgiRJkiRJkiarlw+APwM4u6o2DR1Mcgxwf1WtmsScb0pyM3ALsC/tbYQbgceAC5K8EXh0vEmqamVVtaqqNbDjgkmkIUmSJEmSpMnoWmdWkr2BLcD9o4QcAhyf5IPAzsCTSR4DdgfekOSXgTnA/CR/U1VvHWe9vWh3eB1UVQ8luQiYU1VPJDkYeA1wPHAycOSUH1CSJEmSJEnTrivFrCQLgfOAc6uqRoqpqsOHxJ8BbKqqc5uh9zbjRwCnjVfIaswHfgRsTPJc2gfQX9dsZ9yxqr6Y5Abg7ib+ESa+jVGSJEmSJEkzaDaLWXOTrAa2B54ALqb9psJZUVVrktwC3AncC9zQXNoJ+HySOUCA9zTjnwLOT3IKcHxV3TXSvPvvvoDBFUtnNnlJkiRJkiQBkFEao9ShVqtVg4OD3U5DkiRJkiRpm5FkVVW1RrrWywfAS5IkSZIkST+lawfAb5XkKODMYcP3VNVxE5znCmCvYcOnV9W1U8lPkiRJkiRJvaPrxaym2DTlgtNEi1+SJEmSJEnqP24zlCRJkiRJUt+wmCVJkiRJkqS+YTFLkiRJkiRJfcNiliRJkiRJkvpG1w+A73drN2xk0fKru52GtnHrVyztdgqSJEmSJPUEO7MkSZIkSZLUN2asmJVkS5LVSdYlWZPk1CTjrpfkeUk2JTltyNj6JGub+QZnKmdJkiRJkiT1tpncZri5qpYAJNkVuBSYD7x/nPvOAr40wvirq+oH05qhJEmSJEmS+sqsbDOsqvuBZcDJSTJaXJJjgXuAdZNdK8l1Sc5OMpjkjiQHJflskm8l+dMhcZ9LsqrpHFvWjD2/idslyXZJrk/yS5PNRZIkSZIkSdNr1g6Ar6q7kwwAuwLfH349yTzgdOB1wGnDbwe+nKSA/1NVK8dZ7sdV1UryTuDzwMuAB4G7kpxdVQ8Av11VDyaZC9yU5DNV9Z0kZwIfB74B3F5VXx4h12W0i3MMzF/Y8X8DSZIkSZIkTU0vvc3wDODsqto0QvPWL1bVhma74t8nubOq/nGMua5s/l4LrKuq7wIkuRvYE3gAOCXJcU3cnsA+wANV9Ykkvw6cCCwZafKmmLYSYIfd9qmJPaYkSZIkSZIma9aKWUn2BrYA948ScghwfJIPAjsDTyZ5rKrOraoN0N6umOQK4GBgrGLW483fTw75vPX7s5IcAbwWeEVVPZrkOmBOk+eOwB5N/DzgkQk8piRJkiRJkmbQrBSzkiwEzgPOraoRO5mq6vAh8WcAm6rq3CQ/A2xXVY80n38J+JMpprQAeKgpZC0GXj7k2pnAJcB3gPOBY6a4liRJkiRJkqbJTBaz5iZZDWwPPAFcTPtNhRP1XOCKZuvhs4BLq+qaKeZ2DXBikjuAbwJfB0jyKuAg4LCq2pLk15K8o6ounOJ6kiRJkiRJmgYZpVFKHWq1WjU4ONjtNCRJkiRJkrYZSVZVVWuka9vNdjKSJEmSJEnSZM362wyTHEX7XKqh7qmq40aKH2OejwKHDRs+xy2BkiRJkiRJ265ZL2ZV1bXAtdMwz0nTkI4kSZIkSZL6iNsMJUmSJEmS1DcsZkmSJEmSJKlvWMySJEmSJElS37CYJUmSJEmSpL5hMUuSJEmSJEl9Y9bfZritWbthI4uWX93tNLSNW79iabdTkCRJkiSpJ8x6Z1aS9yVZl+TWJKuTHDJK3CVJvpnktiR/lWT7ZvxXh9w7mOQXx1hrUZLbJpnnH03mPkmSJEmSJM2cWS1mJXkFcAzw0qo6AHgtcO8o4ZcAi4H9gbnA7zbjXwEOrKolwG8Dn5ihdC1mSZIkSZIk9ZjZ3ma4G/CDqnocoKp+MFpgVX1x6+ck3wD2aMY3DQn7GaA6WTjJIuDi5h6Ak6vqn5PsBlwGzKf93+O/AkuBuUlWA+uq6jc7WUOSJEmSJEkza7a3GX4Z2DPJvyT5WJJXjXdDs73wbcA1Q8aOS3IncDXt7qxO3A+8rqpeCrwZ+Mtm/D8D1zadXgcCq6tqObC5qpaMVMhKsqzZ4ji45dGNHS4vSZIkSZKkqZrVYlbTVfUyYBnw78BlSU4Y57aPAf9YVdcPmeeKqloMHAv8zw6X3x44P8la4NPAi5vxm4B3JDkD2L+qHungOVZWVauqWgM7LuhweUmSJEmSJE3VrB8AX1Vbquq6qno/cDLwa6PFJnk/sBB4zyhz/SOwd5JdOlj63cD3aXdftYBnD5njlcAG4KIkvzWBx5EkSZIkSdIsmu0D4F+UZJ8hQ0uA74wS+7vAUcBvVNWTQ8ZfmCTN55cCOwAPdLD8AuC7zVxvAwaaOZ4PfL+qzqd9mPxLm/ifbH2DoiRJkiRJknrDbB8APw/4SJKdgSeAb9PecjiS82gXur7W1K4+W1V/QruT67eS/ATYDLy5qjo5BP5jwGeazqtrgB8140cAf9jMtwnY2pm1Erg1yc0eAC9JkiRJktQb0lkdSKNptVo1ODjY7TQkSZIkSZK2GUlWVVVrpGuzfmaWJEmSJEmSNFmzvc3waZJcAew1bPj0qrp2AnPsD1w8bPjxqjpkqvlJkiRJkiSpd3S9mFVVx03DHGtpHyYvSZIkSZKkbZjbDCVJkiRJktQ3LGZJkiRJkiSpb1jMkiRJkiRJUt+wmCVJkiRJkqS+YTFLkiRJkiRJfaPrbzPsd2s3bGTR8qu7nYb6xPoVS7udgiRJkiRJfa0rnVlJtiRZnWRNkpuTHDpG7PObmNVJ1iU5cTZzlSRJkiRJUu/oVmfW5qpaApDkKODPgVeNEvtd4BVV9XiSecBtSa6sqn+bnVQlSZIkSZLUK3rhzKz5wEOjXayqH1fV483XHRgn5ySbknyo6eL6v0kOTnJdkruTvKGJWZTk+qbj66nOsCTHJflK2nZL8i9J/r9pek5JkiRJkiRNUbc6s+YmWQ3MAXYDjhwrOMmewNXAC4E/HKcr62eA/1dVf5jkCuBPgdcBLwb+GrgSuB94XVU9lmQf4G+BVlVdkeTXgJOAo4H3V9X3RshnGbAMYGD+ws6fWpIkSZIkSVPSC9sMXwF8Msl+VVUjBVfVvcABSX4e+FySy6vq+6PM/WPgmubzWuDxqvpJkrXAomZ8e+DcJEuALcB/GnL/HwC3AV+vqr8dJZ+VwEqAHXbbZ8ScJUmSJEmSNP26vs2wqr4G7AKM2+LUdGTdBhw+RthPhhTFngQeb+59kv8o3r0b+D5wINACnj3k/j2a+56bpOv/fSRJkiRJkvQful6sSbIYGAAeGOX6HknmNp9/FvhF4JtTXHYB8N2mwPW2Zn2SPAv4K+A3gDuA90xxHUmSJEmSJE2jbp+ZBRDg7VW1ZZTYXwD+d5JqYj9cVWunuP7HgM8k+S3aWxJ/1Iz/EXB9Vf1TkjXATUmurqo7prieJEmSJEmSpkFGOaZKHWq1WjU4ONjtNCRJkiRJkrYZSVZVVWuka13fZihJkiRJkiR1qlvbDJ8myf7AxcOGH6+qQ0aJvxHYYdjw26ZhC6IkSZIkSZJ6VM8Us5oi1JIJxI9Y5JIkSZIkSdK2y22GkiRJkiRJ6hsWsyRJkiRJktQ3LGZJkiRJkiSpb1jMkiRJkiRJUt+wmCVJkiRJkqS+0TNvM+xXazdsZNHyq7udhvrE+hVLu52CJEmSJEl9rSc6s5JsSbI6yW1JvpBk5zFin5/k5iZ+XZITZzFVSZIkSZIkdVFPFLOAzVW1pKr2Ax4EThoj9rvAK6pqCXAIsDzJz89CjpIkSZIkSeqyXilmDfU1YPfRLlbVj6vq8ebrDozzDEk+nmSw6eL6QDN2dJJPD4k5IslVzeffSfIvSb6R5Pwk5075iSRJkiRJkjQteqqYlWQAeA1w5Thxeya5FbgXOLOq/m2M8PdVVQs4AHhVkgOA/wsckuRnmpg3A59qOrz+B/By4DBg8SjrL2sKZINbHt04gSeUJEmSJEnSVPRKMWtuktXA94DnAn8/VnBV3VtVBwAvBN6e5LljhL8pyc3ALcC+wIur6gngGuBXkjwLWAp8HjgY+IeqerCqfgJ8eqQJq2plVbWqqjWw44IJPagkSZIkSZImr1eKWZubM7CeD4Sxz8x6StORdRtw+EjXk+wFnAa8pil+XQ3MaS5/CngTcCQwWFWPTOUBJEmSJEmSNPN6pZgFQFU9CpwCnNp0TD1Nkj2SzG0+/yzwi8A3R5lyPvAjYGPTvfX6Idf+AXgp8Hu0C1sAN9Heivizzfq/NsVHkiRJkiRJ0jTqqWIWQFXdAtwK/MYoIb8A3JhkDe2C1Ierau0oc62hvb3wTuBS4IYh17YAV9EucF3VjG0A/hfwjSZ2PeChWJIkSZIkST0iVdXtHHpKknlVtanpzLoC+KuqumK0+FarVYODg7OXoCRJkiRJ0jYuyarmhX5P03OdWT3gjOYw+tuAe4DPdTUbSZIkSZIkPWXEc6l6QZL9gYuHDT9eVYeMEn8jsMOw4beNtgVxNFV12kTiJUmSJEmSNHt6tpjVFKGWTCB+xCKXJEmSJEmSth1uM5QkSZIkSVLfsJglSZIkSZKkvmExS5IkSZIkSX3DYpYkSZIkSZL6hsUsSZIkSZIk9Y2efZthv1i7YSOLll/d7TQ0y9avWNrtFCRJkiRJekaalc6sJFuSrE6yLsmaJKcmGXftJM9LsinJaUPGdk5yeZI7k9yR5BUzm70kSZIkSZJ6xWx1Zm2uqiUASXYFLgXmA+8f576zgC8NGzsHuKaqjk/ybGDHac5VkiRJkiRJPWrWz8yqqvuBZcDJSTJaXJJjgXuAdUPGFgCvBC5o5vpxVT08xhy/l+SmphvsM0l2TLIgyXe2doYl+Zkk9ybZPslBSW5tusg+lOS26XhmSZIkSZIkTY+uHABfVXcDA8CuI11PMg84HfjAsEt7Af8OXJjkliSfSPIzYyz12ao6qKoOBO4AfqeqNgKrgVc1MccA11bVT4ALgf/SdJFtmdTDSZIkSZIkacb06tsMzwDOrqpNw8afBbwU+HhVvQT4EbB8jHn2S3J9krXAbwL7NuOXAW9uPr8FuCzJzsBOVfW1ZvzS0SZNsizJYJLBLY9unMBjSZIkSZIkaSq68jbDJHvT7ny6f5SQQ4Djk3wQ2Bl4MsljwOXAfVV1YxN3OWMXsy4Cjq2qNUlOAI5oxq8E/leSnwNeBvw/YKdO86+qlcBKgB1226c6vU+SJEmSJElTM+vFrCQLgfOAc6tqxEJQVR0+JP4MYFNVndt8vzfJi6rqm8BrgNvHWG4n4LtJtqfdmbWhmX9TkptoHyZ/VVVtAR5O8kiSQ5pi2Vum+qySJEmSJEmaXrNVzJqbZDWwPfAEcDHtNxVOxh8AlzRvMrwbeMcYsf8DuJH2OVs38tPdV5cBn+Y/urUAfgc4P8mTwD8A7iGUJEmSJEnqIRmlOeoZKcm8red0JVkO7FZV7xzrnlarVYODg7OSnyRJkiRJ0jNBklVV1RrpWlfOzOphS5O8l/Z/l+8AJ3Q3HUmSJEmSJA3V1WJWkqOAM4cN31NVx01wno8Chw0bPqeqLpzIPFV1Ge3th5IkSZIkSepBXS1mVdW1wLXTMM9J05COJEmSJEmSetx23U5AkiRJkiRJ6pTFLEmSJEmSJPUNi1mSJEmSJEnqGxazJEmSJEmS1DcsZkmSJEmSJKlvdPVthtuCtRs2smj51d1OQ7Ns/Yql3U5BkiRJkqRnJDuzJEmSJEmS1Dd6ppiVZEuS1UluS/LpJDuOEjcnyTeSrEmyLskHxpn3uiStSeRzbJIXT/Q+SZIkSZIkzZyeKWYBm6tqSVXtB/wYOHGUuMeBI6vqQGAJcHSSl89APscCFrMkSZIkSZJ6SC8Vs4a6HnjhSBeqbVPzdfvmT3UyaZKPJxkc3tGVZEWS25PcmuTDSQ4F3gB8qOkWe8HUHkeSJEmSJEnToecOgE/yLOD1wDVjxAwAq2gXvD5aVTd2OP37qurB5v6vJDkA2AAcByyuqkqyc1U9nORK4KqqunyE9ZcBywAG5i+cyONJkiRJkiRpCnqpM2tuktXAIPCvwAWjBVbVlqpaAuwBHJxkvw7XeFOSm4FbgH1pbyPcCDwGXJDkjcCj401SVSurqlVVrYEdF3S4tCRJkiRJkqaqlzqzNjcFqo41HVRfBY4GbhsrNslewGnAQVX1UJKLgDlV9USSg4HXAMcDJwNHTiJ/SZIkSZIkzbBe6szqSJKFSXZuPs8FXgfc2cGt84EfARuTPJf2VkaSzAMWVNUXgXcDBzbxjwA7TW/2kiRJkiRJmope6szq1G7AXzfnXm0H/F1VXTXeTVW1JskttAtf9wI3NJd2Aj6fZA4Q4D3N+KeA85OcAhxfVXeNNO/+uy9gcMXSKT2QJEmSJEnSWH7yk59w33338dhjj3U7lWk1Z84c9thjD7bffvuO70lVRy8C1CharVYNDg52Ow1JkiRJkrQNu+eee9hpp514znOeQ5JupzMtqooHHniARx55hL322uunriVZVVWtke7ru22GkiRJkiRJzzSPPfbYNlXIAkjCc57znAl3m/XsNsMkzwG+MsKl11TVAyPEXwHsNWz49Kq6dibykyRJkiRJmk3bUiFrq8k8U88Ws5qC1ZIJxB83c9lIkiRJkiQ9sx166KH88z//c7fT6N1iliRJkiRJkka2aPnV0zrf+g5ebtcLhSzwzCxJkiRJkiR1YN68eQBcd911vOpVr+JXf/VX2XvvvVm+fDmXXHIJBx98MPvvvz933XUXACeccAInnngirVaL//Sf/hNXXXXVtORhZ5YkSZIkSZImZM2aNdxxxx383M/9HHvvvTe/+7u/yze+8Q3OOeccPvKRj/AXf/EXAKxfv55vfOMb3HXXXbz61a/m29/+NnPmzJnS2nZmSZIkSZIkaUIOOuggdtttN3bYYQde8IIX8Eu/9EsA7L///qxfv/6puDe96U1st9127LPPPuy9997ceeedU17bzqwpWrth47TvU9Xs6WRPsCRJkiRJ+mk77LDDU5+32267p75vt912PPHEE09dG/62wul4I6OdWZIkSZIkSZoRn/70p3nyySe56667uPvuu3nRi1405TlnrDMryRZgLbA98ATwSeDsqnpylPhFwB3AN5uhr1fVic21lwEXAXOBLwLvrKqaqdwlSZIkSZI0dc973vM4+OCD+eEPf8h555035fOyYGa3GW6uqiUASXYFLgXmA+8f4567tt4zzMeB3wNupF3MOhr40nQmK0mSJEmS1C+6cWzOpk2bADjiiCM44ogjnhq/7rrrnvo8/NprX/tazjvvvGnNY1a2GVbV/cAy4ORMcHNkkt2A+VX19aYb65PAsWPEX5fk7CSDSe5IclCSzyb5VpI/HRL3uSSrkqxLsqwZe34Tt0uS7ZJcn+SXJvPMkiRJkiRJmn6zdgB8Vd2dZADYFfj+KGF7JbkF+CHw36vqemB34L4hMfc1Y2P5cVW1krwT+DzwMuBB4K4kZ1fVA8BvV9WDSeYCNyX5TFV9J8mZtDvBvgHcXlVfHj55U/xaBjAwf2Fn/wEkSZIkSZKeQS666KIZmbeX3mb4XeB5VfVAc0bW55LsO8m5rmz+Xgusq6rvAiS5G9gTeAA4JclxTdyewD7AA1X1iSS/DpwILBlp8qpaCawE2GG3fTy7S5IkSZIkaZbMWjEryd7AFuD+ka5X1ePA483nVUnuAv4TsAHYY0joHs3YWB5v/n5yyOet35+V5AjgtcArqurRJNcBc5o8dxyy3jzgkfGfTpIkSZIkaWZVFRM8vannTeb9frNyZlaShcB5wLmjvYUwycJmG+LWwtc+wN1NV9UPk7y8OW/rt2hvHZyKBcBDTSFrMfDyIdfOBC4B/hg4f4rrSJIkSZIkTdmcOXN44IEHJlX86VVVxQMPPDDhNxzOZGfW3CSrge2BJ4CLgbPGiH8l8CdJfkK7g+rEqnqwufb7wEXAXNpvMZzqmwyvAU5McgfwTeDrAEleBRwEHFZVW5L8WpJ3VNWFo020/+4LGOzCGwQkSZIkSdIzxx577MF9993Hv//7v3c7lWk1Z84c9thjj/EDh8i2VNHrhlarVYODg91OQ5IkSZIkaZuRZFVVtUa6NivbDCVJkiRJkqTpMOtvM0xyFO1zqYa6p6qOGyl+jHk+Chw2bPicsbYESpIkSZIkqb/NejGrqq4Frp2GeU6ahnQkSZIkSZLURzwza4qSPEL7EHmpE7sAP+h2Euob/l40Ef5eNBH+XtQpfyuaCH8vmgh/LxrP86tq4UgXZr0zaxv0zdEOJJOGSzLo70Wd8veiifD3oonw96JO+VvRRPh70UT4e9FUeAC8JEmSJEmS+obFLEmSJEmSJPUNi1lTt7LbCaiv+HvRRPh70UT4e9FE+HtRp/ytaCL8vWgi/L1o0jwAXpIkSZIkSX3DzixJkiRJkiT1DYtZU5Dk6CTfTPLtJMu7nY96S5K/SnJ/ktuGjP1ckr9P8q3m75/tZo7qDUn2TPLVJLcnWZfknc24vxc9TZI5Sb6RZE3ze/lAM75Xkhubf5MuS/Lsbueq3pFkIMktSa5qvvt70YiSrE+yNsnqJIPNmP8eaURJdk5yeZI7k9yR5BX+XjSSJC9q/ndl658fJnmXvxdNlsWsSUoyAHwUeD3wYuA3kry4u1mpx1wEHD1sbDnwlaraB/hK8116Aji1ql4MvBw4qfnfE38vGsnjwJFVdSCwBDg6ycuBM4Gzq+qFwEPA73QvRfWgdwJ3DPnu70VjeXVVLamqVvPdf480mnOAa6pqMXAg7f+d8feip6mqbzb/u7IEeBnwKHAF/l40SRazJu9g4NtVdXdV/Rj4FPCrXc5JPaSq/hF4cNjwrwJ/3Xz+a+DY2cxJvamqvltVNzefH6H9fwjujr8XjaDaNjVft2/+FHAkcHkz7u9FT0myB7AU+ETzPfh70cT475GeJskC4JXABQBV9eOqehh/Lxrfa4C7quo7+HvRJFnMmrzdgXuHfL+vGZPG8tyq+m7z+XvAc7uZjHpPkkXAS4Ab8feiUTRbxlYD9wN/D9wFPFxVTzQh/pukof4C+G/Ak8335+DvRaMr4MtJViVZ1oz575FGshfw78CFzTbmTyT5Gfy9aHxvAf62+ezvRZNiMUvqkmq/StTXieopSeYBnwHeVVU/HHrN34uGqqotTZv+HrQ7hRd3NyP1qiTHAPdX1apu56K+8YtV9VLaR2mclOSVQy/675GGeBbwUuDjVfUS4EcM2yLm70XDNWc0vgH49PBr/l40ERazJm8DsOeQ73s0Y9JYvp9kN4Dm7/u7nI96RJLtaReyLqmqzzbD/l40pmY7x1eBVwA7J3lWc8l/k7TVYcAbkqynfSTCkbTPuPH3ohFV1Ybm7/tpn2dzMP57pJHdB9xXVTc23y+nXdzy96KxvB64uaq+33z396JJsZg1eTcB+zRvA3o27VbJK7uck3rflcDbm89vBz7fxVzUI5rzay4A7qiqs4Zc8veip0myMMnOzee5wOton7P2VeD4JszfiwCoqvdW1R5VtYj2/63y/6rqN/H3ohEk+ZkkO239DPwScBv+e6QRVNX3gHuTvKgZeg1wO/5eNLbf4D+2GIK/F01S2p18mowkv0z7HIoB4K+q6s+6m5F6SZK/BY4AdgG+D7wf+Bzwd8DzgO8Ab6qq4YfE6xkmyS8C1wNr+Y8zbf6I9rlZ/l70U5IcQPuA1AHa/0+pv6uqP0myN+3Om58DbgHeWlWPdy9T9ZokRwCnVdUx/l40kuZ3cUXz9VnApVX1Z0meg/8eaQRJltB+ucSzgbuBd9D824S/Fw3TFMn/Fdi7qjY2Y/7viybFYpYkSZIkSZL6htsMJUmSJEmS1DcsZkmSJEmSJKlvWMySJEmSJElS37CYJUmSJEmSpL5hMUuSJEmSJEl9w2KWJEmSJEmS+obFLEmSJEmSJPUNi1mSJEmSJEnqG/8/Bkfq0YINPYMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1440x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# feature importance for top 30 features\n",
    "fea_imp = pd.DataFrame({'imp':model.feature_importances_, 'col': features})\n",
    "fea_imp = fea_imp.sort_values(['imp', 'col'], ascending=[True, False]).iloc[-30:]\n",
    "_ = fea_imp.plot(kind='barh', x='col', y='imp', figsize=(20, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "block:evaluation_result",
     "prev:modelling"
    ]
   },
   "outputs": [],
   "source": [
    "model = joblib.load('lgb.jl')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "binary_logloss = model.booster_.best_score.get('valid_0').get('binary_logloss')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "amex_metric_score = model.booster_.best_score.get('valid_0').get('amex_metric_score')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": [
     "pipeline-metrics"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.24605493989573005\n"
     ]
    }
   ],
   "source": [
    "print(binary_logloss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": [
     "pipeline-metrics"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7625788091718922\n"
     ]
    }
   ],
   "source": [
    "print(amex_metric_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "sub = pd.DataFrame({'customer_ID': test.index,\n",
    "                    'prediction': np.mean(y_pred_list, axis=0)})\n",
    "sub.to_csv('submission.csv', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "skip"
    ]
   },
   "outputs": [],
   "source": [
    "sub"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kubeflow_notebook": {
   "autosnapshot": true,
   "experiment": {
    "id": "2efb8e27-3b2e-439b-a53c-b1f9d7b94cfc",
    "name": "g-research-crypto-forecasting"
   },
   "experiment_name": "g-research-crypto-forecasting",
   "katib_metadata": {
    "algorithm": {
     "algorithmName": "grid"
    },
    "maxFailedTrialCount": 3,
    "maxTrialCount": 12,
    "objective": {
     "objectiveMetricName": "",
     "type": "minimize"
    },
    "parallelTrialCount": 3,
    "parameters": []
   },
   "katib_run": false,
   "pipeline_description": "forecasting short term returns in 14 popular cryptocurrencies.",
   "pipeline_name": "g-research-crypto-forecasting-pipeline",
   "snapshot_volumes": true,
   "steps_defaults": [
    "label:access-ml-pipeline:true",
    "label:kaggle-secret:true",
    "label:access-rok:true"
   ],
   "volume_access_mode": "rwm",
   "volumes": [
    {
     "annotations": [],
     "mount_point": "/home/jovyan",
     "name": "test-workspace-qtvmt",
     "size": 32,
     "size_type": "Gi",
     "snapshot": false,
     "type": "clone"
    }
   ]
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
